Electronic image stabilization apparatus

ABSTRACT

A video signal processing apparatus includes circuitry which generates a motion vector which represents the amount and direction of motion which occurs between consecutive frames of a video signal and for determining if the vector indicates that an image sensing device is undergoing undesirable motion. The circuitry is further arranged to produce a correction signal based on the motion vector and delay the video signal in accordance with the same. A memory is provided in which video information relating to a peripheral portion of the effective picture area of the image sensing device, is stored and operatively connected with a selector which selects one of the delayed video signal and peripheral portion data stored in the memory in accordance with the correction signal.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to a video signal processing apparatus applicableto a manually induced vibration correction device for video data such asan output produced by a hand-held type video camera.

2. Description of the Prior Art

When a picture is taken using a portable hand-held or so called "handy"type video camera, there is a problem that the reproduced picture jumpsabout or vibrates due to vibration caused by manual manipulation of thedevice. To solve this problem, a technique where a motion vector isdetected and video data stored in a picture memory is then correctedbased on this motion vector, has been proposed (for example, JapanesePatent Disclosure Sho 63-166370). The detection of the motion vector isachieved using block matching, for instance. With this technique, apicture is divided into many areas (called "blocks"), an absolute valueof a frame difference between a representative point of a previous framelying at a central portion of each block and picture element data withina block of the present frame is calculated, an absolute value of a framedifference is accumulated with respect to one picture, and the motionvector is detected from the position of a minimum value of accumulatedframe difference data. Also, as described in this publication, pictureenlargement to the extent of 15%, for example, is carried out byread-out control of the picture memory, interpolation circuit is used toprevent video data from being lost when the motion correction is made.For such a manually induced vibration correction device, accuratedetection of the manually induced vibration is required.

Enlargement of a picture is executed by a process which delays theread-out speed of a picture memory as compared with its write-in speedand which interpolates deficient picture element data. The pictureenlargement consequently causes deterioration such as blurring ascompared with the original picture.

OBJECTS AND SUMMARY OF THE INVENTION

It is, therefore, an object of the invention to provide a manuallyinduced vibration correction device for video data which eliminates theneed for picture enlargement and which is capable of preventing picturequality deterioration.

According to an aspect of the Present invention there is provided avideo signal processing apparatus for processing a video signalgenerated by an image sensing device in a video camera comprising:

a circuit for generating a motion vector representing the amount and thedirection of the motion which occurs between consecutive frames of thevideo signal,

a circuit for determining whether the motion vector represents anundesired motion of the image sensing device or not,

a correction signal generating circuit for generating a correctionsignal according to the motion vector,

a delay circuit for delaying the video signal in response to thecorrection signal,

a memory circuit for storing a video information corresponding to aperipheral portion of an effective picture area of the image sensingdevice, and

a selector for selecting one of the outputs of the memory and the delay,and for causing a corrected video signal to be generated on the basis ofthe selection.

The above, and other, objects, features and advantages of the presentinvention will become readily apparent from the following detaileddescription thereof which is to be read in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an embodiment of the presentinvention;

FIG. 2 is a schematic diagram showing a block division which is usedwith the present invention;

FIG. 3 is a schematic diagram showing a motion vector search range usedin the embodiment of this invention;

FIG. 4 is a schematic diagram depicting the formation of a frequencydistribution table which is used in accordance with of this invention;

FIGS. 5A and 5B are schematic diagrams showing the correction ofmanually induced vibration in accordance with this invention; and

FIGS. 6A and 6B are a schematic diagrams depicting the display of themanually induced vibration as it appears in the view finder of a videocamera.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereunder, an embodiment of this invention will be described withreference to the drawings. In this embodiment, it should be noted thatmanually induced vibration results in a movement of a video camera, andthe movement of the picture as a whole. However, while the movement ofan object means that a certain portion of a picture moves, there is astill portion in the picture, which is the same and this allowsdistinction between the two to be performed. In FIG. 1, 1 is an inputterminal for digital video data. The video data is not limited to thatproduced by a CCD of the video camera, for example, but can also includea sequence of the order of interlaced scanning.

The input video data is supplied to a field memory 2 and arepresentative point memory 3. Data which has been subjected tocorrection for manually induced vibration by a correction signal(described later) is read out at an output of the field memory 2. Whenthe processing is done on a frame by frame basis, a frame memory is usedin place of the field memory 2. The output of the representative pointmemory 3 is supplied to a subtracter 4 and a representative point memory5. The output of the representative point memory 5 is supplied asubtracter 6. The input video data is supplied to the subtracters 4 and6. The representative point memory 3 stores data of the representativepoints of the previous field, while the representative point memory 5stores data of the representative points two fields before (i.e., oneframe).

As shown in FIG. 2, a picture of one field is segmented into blocks of mpicture elements×n lines, and a central picture element of each block istaken as a representative point. The representative points aredistributed evenly over the picture. The subtractor 4 detects respectivedifferences (that is, inter-field differences) between respective dataof m×n picture elements of a certain block in the present field and dataof a representative point of a block at the same position in a previousfield. The inter-field difference is supplied to an absolute valueintegration circuit 7. Similarly, the subtracter 6 detects a framedifference between a block of the present field and a representativepoint lying before two fields. The frame difference is fed to anabsolute value integration circuit 8 and a frequency distribution tablegenerator 9.

The detection of a motion vector is made using the frame difference atthe absolute value integration circuit 8. The inter-field movementprovided by the absolute value integration circuit 7 from theinter-field difference is used as an auxiliary means. Specifically,there is a feature that the inter-field difference provides goodaccuracy in terms of time but poor accuracy in terms of position, whilethe inter-frame difference provides poor accuracy in terms of time butgood accuracy in terms of position. As a result, when a motion vector isdetected from the frame difference, detection accuracy can be improvedusing the inter-field difference in combination.

As shown in FIG. 3, the search area of movement is handled in the samemanner as the block size. m×n differences between a representative pointlying before two fields and picture element data within a block of thepresent field are produced at the subtracter 6 for each block. In theabsolute value integration circuit 8, the absolute value of a framedifference of each position within the block is accumulated over oneframe period, distribution of accumulated frame difference data of m×nis formed, and a minimum value in the distribution is detected as amotion vector. The detection of the motion vector is carried out in theconventional manner.

A frame difference from the subtracter 6 and the input video data aresupplied to the frequency distribution table generator 9. In thefrequency distribution table generator 9, as shown in FIG. 4, a memoryhaving addresses corresponding to eight Peripheral picture elements a,b, c, d, e, f, g, and h of a picture element x, which coincides with thecenter of the block of FIG. 3, is provided. Among these pictureelements, the frequency of positions (addresses of the memory) having aninclination in terms of space, is updated each time the frame differencebecomes zero. For example, when the frame difference is almost zero andwhen an absolute value of a difference between a value of the pictureelement x and a value of the picture element d is greater than athreshold value, the address of the memory corresponding to the positionof d is brought to +1. This processing for each block is accumulatedover the whole picture to provide a frequency distribution table. Whenthe inclination in terms of space lies in the horizontal direction(between the picture element x and the picture element e or between thepicture element x and the picture element d), for example, a framedifference occurs without fail as a result of horizontal movement.Consequently, when the above-mentioned conditions are met, it means thatits block is not moving that direction.

A moving vector detected at the absolute value integration circuit 8 andthe output of the frequency distribution table generator 9 are suppliedto a motion amount decision circuit 10. The motion amount decisioncircuit 10 verifies whether the motion vector can be employed as amanually induced vibration vector. Specifically, it decides that themotion vector is generated from the movement of the whole picture whenthe frequency of positions corresponding to the moving picture, that is,the portion of still blocks, is less than a threshold value fordecision. Simultaneously, it decides that the moving vector is amanually induced vector. Otherwise, it decides that there are manystationary blocks in the picture and that the motion vector is generatedby the movement of an object in the picture. For example, when thusdetected motion vector is in the upward or vertical direction, thefrequency of the position of the picture element b is examined. If thisfrequency is larger than a threshold level, it is understood that thereare many stationary portions in the picture and that the motion vectoris not a manually induced vibration vector.

The supply of the inter-field moving vector from the absolute valueintegration circuit 7 to the moving amount decision circuit 10 is tocheck the stability of a detected moving vector and to improve theaccuracy. The output (manually induced vibration vector) of the movingamount decision circuit 10 is supplied to a correction amount generator11. The manually induced vector is detected from the inter-framemovement but is not the same as the correction amount for hand-causedvibration. For instance, when manually induced vibration in the samedirection takes place over three consecutive frame periods, a manuallyinduced vibration vector V1 between the first and next frames isdetected, and a manually induced vibration vector V2 between the secondand third frames is detected. Although a correction amount for thesecond frame may be V1, a correction amount for the third frame must be(V1+V2). The correction amount generator 11 generates a correctionamount provided by the integration of the manually induced vector.Further, as in this embodiment, when manually induced vibrationcorrection is made at the unit of field, a motion vector is brought to1/2 to correspond to the time difference of the field and frame for theconversion of an inter-frame motion amount into an inter-field one.

A manually induced vibration correction signal from the correctionamount generator 11 is supplied to an address control circuit 12 and aselection signal generator 13. The address control circuit 12 generatesaddress signals for the field memory 2 and a peripheral memory 16. Apicture of one field is written into the field memory 2, and itsread-out address is controlled depending on a correction amount.Therefore, video data provided by moving the input video data of onefield depending on the correction amount is obtained from the fieldmemory 2. The output of the field memory 2 is supplied to a selector 14.The output of the selector 14 appears on at an output terminal 15 and isalso supplied to a peripheral memory 16. Peripheral data which is theoutput of the peripheral memory 16 is supplied to the selector 14. Theselector 14 selects video data, which has been subjected to thecorrection of manually induced vibration, from the field memory 2 andperipheral data stored in the peripheral memory 16 in response to aselection signal from the selection signal generator 13.

As shown in FIG. 5, a peripheral portion 22 (area outside of a one-dotand dash line) of one field picture (its picture frame is indicated at21) taken into the field memory 2 is stored in the peripheral memory 16.The width of the peripheral portion 22 is set in consideration of therange of correction of manually induced vibration, for instance, set atthe width of the extent of 10 to 20 percent. In FIG. 5B, when a picturewhich should lie in a position of FIG. 5A, moves due to manually inducedvibration in the right-handed direction with respect to the drawing, forexample, as shown by the picture frame 21, the whole picture iscorrected to a position indicated at a broken line by the manuallyinduced vibration correction amount. In such a case, because of itsabsence in an initially taken picture, a picture portion 23 indicated bythe hatching on the left-hand side of the moved picture is excluded. Theexcluded portion 23 is replaced by a picture at a corresponding positionstored in the peripheral memory 16. This replacement is carried out bythe address control of the address control circuit 12 and the switchingoperation of the selection 14. Also, peripheral picture data other thanthe dropped portion 23 in the taken video data is written into theperipheral memory 16 to update the content of the memory 16.

In this invention, the correction of hand-manually induced vibration ofthe output of a video camera may be carried out on a real-time basis.Further, such correction of the manually induced vibration may beapplied to the output of a VTR. In correcting the output of the videocamera on a real-time basis, it is desirable to display a markerindicative of the presence of manually induced vibration on a picture ofa view finder.

As shown in FIG. 6A, a picture frame indicated by a broken line is thepicture actually taken and it is assumed that this picture has beencorrected by the above manually induced vibration correction as shown ata picture 21b indicated by a solid line. In this case, as shown in FIG.6B, a marker 25 indicated by a vertically extending line is displayed ata position corresponding to an edge of the picture frame 21 of thepicture which is actually being taken. This marker 25 allows the personwho is taking the picture to understand the direction and amount ofmanually induced vibration or the direction and amount of manuallyinduced vibration correction and allow the corrective movement of theview finder in a direction which corrects the vibration. Since afeedback is performed based on the eyesight of a video camera operator,in addition to the above-mentioned correction of the manually inducedvibration, the situation where the range of the manually inducedvibration exceeds the correction range, can be prevented beforehand. Ifthe manually induced vibration is not displayed on the view finder, apicture may move abruptly at a certain time point when such a correctioncannot be carried out. The marker 25 is not limited to a vertical orlongitudinal line and can be an arrow or the like.

Since this invention replaces an excluded portion using data stored inthe peripheral memory, no deterioration such as picture blur takes placeas compared to picture enlargement. With this invention, discontinuityoccurs at a part of the periphery of the picture in terms of time.However, because this is the peripheral portion discontinuity is limitedto the periphery of the picture it is not particularly apparent to theeye of the observer.

Having described a specific preferred embodiment of the presentinvention with reference to the accompanying drawings, it is to beunderstood that the invention is not limited to that precise embodiment,and that various changes and modifications may be effected therein byone skilled in the art without departing from the scope or the spirit ofthe invention as defined in the appended claims.

What is claimed is:
 1. A video signal processing apparatus forprocessing a video signal generated by an image device in a video cameracomprising:means for generating a motion vector representing a motionamount and a motion direction between consecutive frames of said videosignal; means for judging whether said motion vector represents theundesired motion of said image sensing device or not; correction signalgenerating means for generating a correction signal in accordance withsaid motion vector; delay means for delaying said video signal inresponse to said correction signal; memory means for storing a videoinformation corresponding to a peripheral portion of an effectivepicture area of said image sensing device; and selection means,responsive to said correction signal, for selecting at least one of theoutputs of said memory means and said delay means, and for outputting avideo signal.
 2. A video signal processing apparatus according to claim1, wherein said motion vector generating means includes means forstoring pixel data representative of each of a plurality of blocks whicheach have a plurality of pixels, subtracting means for subtractingcurrent corresponding pixel data representative of each of saidplurality of blocks from said stored pixel data, and accumulating meansfor accumulating the outputs of said subtracting means.
 3. A videosignal processing apparatus according to claim 2, wherein said judgingmeans includes detecting means for detecting the difference between thepixel data representative of each block stored in said storing means andthe data representative of the corresponding block in the current frameand distribution table generating means for generating a distributiontable in response to the output of said detecting means, saiddistribution table generating means including a memory having aplurality of address areas corresponding to the pixel of a block.
 4. Avideo signal processing apparatus according to claim 1, wherein saidvideo camera includes a view finder on which a marker corresponding tosaid correcting signal is displayed.